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ABSTRACT 

This policy brief takes a close look at the ways in which the 
Third International Mathematics and Science Study (TIMSS) -1995 and TIMSS-1999 
data have helped to inform changes in policy and practice as schools, 
districts, and states respond to the call for improvement in mathematics and 
science achievement. This policy brief is based primarily on data collected 
in structured interviews with administrators and teachers in 10 TIMSS 
benchmarking jurisdictions (referred to as "benchmarkers ” ) . These 
jurisdictions within the United States participated in the TIMSS-1999 
Benchmarking study, committing their own resources and time to receive data 
from a representative sample of their own eighth-grade students. This brief 
is designed to facilitate networking and continued learning from TIMSS; it 
focuses on benchmarkers* experiences but is relevant for anyone interested in 
using TIMSS to improve mathematics and science instruction. Strategies are 
reported in summary form. Actual TIMSS data and analyses are not discussed in 
detail, but related references are provided at the end of the brief. 

(Contains 16 references and 25 endnotes.) (Author) 
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The Third International Mathen\atics and 
Science Study (TIMSSV1995 and its successor, 
TIMSS'1999, provide education researchers, 
policymakers, and practitioners with rich, 
comparative data designed to help better 
understand the performance of educational 
systems.^ As such, TIMSS is a valuable tool in 
current efforts to improve mathematics and 
science instruction and to educate students in 
the United States to global standards of excel- 
lence. What can we learn from this ambitious 
and unprecedented international effort to 
provide meaningful, useful data for the 
reform of mathematics and science instruc- 
tion? It is important to reflect on this question 
as we assess the promise and challenges of 
using TIMSS-type data in particular, as well 
as the broader national effort to use data to 
guide school improvement in general. 

The first in CPRE's series of Policy Briefs 
about TIMSS (Dunson, 2000) looked at initial 
efforts to make use of TIMSS-1995 data. In 
this Policy Brief, we take a closer look at the 
ways in which TIMSS-1995 and TIMSS-1999 
data have helped to inform changes in policy 
and practice as schools, districts, and states 
respond to the call for improvement in math- 
ematics and science achievement. This Poli- 
cy Brief was prepared to complement CPRE's 
effort to address this question in a TIMSS Pol- 
icy Forum, held in Washington, DC, in May 
2002. This forum convened TIMSS Bench- 
marking jurisdiction representatives, teach- 
ers, administrators, policymakers, research- 
ers, and technical assistance providers to 
share successful strategies and ongoing chal- 
lenges in taking full advantage of TIMSS 
data. 

This Policy Brief is based primarily on 
data collected in structured interviews with 



administrators and teachers in 10 TIMSS 
Benchmarking jurisdictions (referred to as 
"Benchmarkers"). These jurisdictions within 
the United States participated in the TIMSS- 
1999^ Benchmarking Study, committing their 
own resources and time in order to receive 
data from a representative sample of their 
own eighth-grade students. TIMSS Bench- 
markers thus have international comparative 
data on their students' achievement and their 
system variables. Our purposive sample 
included equal representation from states, 
districts, and consortia with a variety of 
demographic characteristics.^ While TIMSS 
data have been used nationwide. Bench- 
marking jurisdictions are notable for their 
existing commitment to reform of mathemat- 
ics and science programs and for the fact that 
they have access to their own local TIMSS- 
1999 data. 

This Brief is designed to facilitate net- 
working and continued learning from 
TIMSS; it focuses on Benchmarkers' experi- 
ences, but is relevant for anyone interested in 
using TIMSS to improve mathematics and 
science instruction. Strategies are reported in 
summary form. Actual TIMSS data and 
analysis are not discussed in detail, but relat- 
ed references are provided at the end of this 
Brief. 

Background: What Do We 
Know About Using Data to 
Drive System and School 
Improvement? 

In the context of standards-based reform, 
data are critical tools in shaping school 
improvement efforts based on systematic use 
of evidence. Research has demonstrated 
great variability, however, in the ways that 
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data have driven reform in different contexts. 
Two broad factors shape this variability: 1) 
the nature and quality of the data, and 2) the 
capacities of systems to analyze and make 
use of data. 

Quality of Data 

Data can be seen as '"scripts'" that guide 
changes in policy and practice (Massell, 
2001). The form and the content of data influ- 
ence their use. Assessments, for example, 
vary in the populations they sample, the unit 
to which scores can be disaggregated, the 
content domains and skill sets assessed, the 
frequency and timing of data reporting, and 
the scaling of scores. These variations affect 
the way data are used to inform district- and 
school-level decision making and classroom 
instruction (0"Day, in press; Herman, in 
press). 

TIMSS has a number of advantages as a 
tool for improvement. First, it was a rigor- 
ously executed international study and there- 
fore has credibility for a variety of stakehold- 
ers looking to measure student achievement 
by global standards. Comparative interna- 
tional data allow educators to consult the 
policies and practices of higher achieving 
nations as possible strategies for reform. Also 
critical is the fact that TIMSS provides data 
not only on student achievement but on as 
many as 1,500 contextual variables that might 
help to explain variations in achievement. By 
supplementing assessment questions with 
student, teacher, and school questionnaires, 
videotaped lessons, and analyses of national 
curriculum frameworks, TIMSS is able to 
provide extensive data relevant to education- 
al policy and practice.^ 

The many forms and broad dissemina- 
tion^ of TIMSS data helped gain attention 
from a variety of stakeholders: the highly 
publicized ranking of international student 
achievement lent urgency and credibility to 
existing critiques of mathematics and science 
instruction in the United States; TIMSS- 
released items helped convince educators 
that the content and skills assessed were rele- 
vant to their students; the videotape data 
helped administrators, teachers, students, 
and the public to see concrete differences in 
lesson content and pedagogical style between 
American, German, and Japanese class- 
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rooms; and TIMSS analyses greatly increased 
the influence of the data by indicating clear, 
actionable alternatives for policy and practice 
— for example, the oft-repeated phrase from 
TIMSS researcher Bill Schmidt's studies that 
curricula in the United States are "a mile 
wide and an inch deep" (Schmidt, McKnight, 
& Raizen, 1997). 

Because TIMSS is not able to provide 
much of the information available from most 
state and local assessments — specifically 
data over time^ or disaggregated data — it 
should be viewed as a useful resource to be 
used in complement with these assessments 
and additional system data. Due to sampling 
procedures, TIMSS results cannot be disag- 
gregated to particular units below the coun- 
try or jurisdiction originally assessed; thus, 
TIMSS-1995 provides no local data^ and 
TIMSS-1999 cannot provide Benchmarkers 
with particular student, school, or, in the case 
of consortia, district data. In some cases these 
limitations presented significant challenges 
for use and interpretation, especially for 
those educators accustomed to the kind of 
particular school- and student-level data 
analysis prevalent in the context of current 
accountability policies. 

System Capacities 

While achievement data can do much to 
shed light on existing problems in student 
performance, they cannot offer explanations 
and solutions for these problems. Data will 
lead to improvements in student achieve- 
ment only if states, districts, and schools have 
the capacity to analyze and use them. 
Research indicates that, in this respect, orga- 
nizations vary greatly along a number of 
dimensions including: organizational needs 
and priorities, qualifications of personnel 
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and technological expertise, existing capacity 
of teachers and administrators collectively to 
implement changes, and level of support for 
professional development (Massell, 2001; 
Abelmann & Elmore, with Even, Kenyon, & 
Marshall, 1999; O'Day, in press; Siskin, in 
press). Research indicates further that, in the 
case of schools, those with higher socioeco- 
nomic status and student achievement are 
more successful at taking advantage of data 
and resources than lower performing schools 
(Elmore, 2001; Siskin, in press). 

Because of their amount and complexity, 
TIMSS-1999 Benchmarking data presented an 
even greater challenge for analysis and use 
than most assessment data. Benchmarkers 
had access to a number of external resources 
to help them in this task. All Benchmarkers 
were offered data analysis training by the 
International Study Center at Boston College 
and data analysis support through Michigan 
State University, fimded by the National Sci- 
ence Foundation. In addition, a number of 
Benchmarkers, through their involvement 
with TIMSS, established ongoing relation- 
ships with researchers, universities, regional 
educational laboratories, other Benchmark- 
ers, and national organizations such as the 
American Academy for the Advancement of 
Science (Project 2061, in particular), the 
National Science Foundation, and the 
National Goals Panel. Though some have 
been better able to access TIMSS-associated 
opportunities than others, all sampled Bench- 
markers expressed great enthusiasm for the 
networking possibilities represented by 
TIMSS and a frustration at lack of sufficient 
time and resources to take full advantage of 
them. 

Benchmarkers’ Experience 
Using TIMSS to Inform Policy 
and Practice at the Local Level 

Setting the Agenda 

TIMSS data have been influential in both 
direct and indirect ways. In many cases, 
TIMSS data led directly to the formation of 
new initiatives such as Ohio's SMART (Sci- 
ence and Math Achievement Required for 
Tomorrow) Consortium, established explicit- 
ly to address mathematics and science 
achievement using higher standards, greater 



focus and rigor, and internationally competi- 
tive achievement goals. In other cases, 
TIMSS' influence was less direct, but, as 
many Benchmarking representatives indicat- 
ed, there is little doubt that it has, over time, 
become part of the working language with 
which educators and the public think about 
mathematics and science instruction. As a 
representative from the Math and Science 
Collaborative® explained, "TIMSS informs 
virtually everything that we do in one way or 
another, either in very explicit conversations 
about TIMSS or in implicit ways, supporting 
the direction we are moving in." 

TIMSS Highlighted Issues for Reform 

Student Achievement. The majority of 
Benchmarkers looked at their own TIMSS- 
1999 data primarily to compare student 
achievement in specific subtopics to other 
national and international jurisdictions. 
These results generally verified what Bench- 
markers had learned from national analyses 
of TIMSS-1995 data and their own assess- 
ments of student achievement — that, for 
example, their students are weaker in physi- 
cal science, inquiry, and measurement than 
their international peers. These results were 
not surprising for most, but they helped con- 
vince educators and the public that conclu- 
sions from national-level data had local rele- 
vance. In some cases, student achievement in 
certain subtopics was not as high as expected, 
leading to reform of existing programs.’ 
Many Benchmarkers also did comparisons of 
student achievement on specific released 
items; this was especially useful in helping 
mathematics and science leadership to think 
about results in relationship to their specific 
curricula. 

Exploring Possible Relationships 
Between Contextual Variables and Achieve- 
ment. TIMSS-1995 and TIMSS-1999 did more 
than focus attention on the need for reform — 
they initiated a whole new level of discus- 
sions not possible with achievement data 
alone. TIMSS data raised questions about 
specific components of policy and practice, 
suggesting, most importantly, that concrete 
steps could be taken to improve instruction 
and increase achievement. 
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Benchmarkers noted the strong influence 
of general messages emerging from analyses 
of contextual variables in TIMSS 1995 data — 
specifically, the lack of curricular coherence 
and rigor in the United States, the need for 
increased focus on inquiry and problem solv- 
ing, and the need to increase teachers' con- 
ceptual and pedagogical knowledge. While 
these initial findings did not indicate estab- 
lished relationships between the system vari- 
ables mentioned and student achievement, 
they highlighted important differences 
between policies and practice in the United 
States and many high-achieving nations.’*^ 
This provided policymakers and practition- 
ers with specific questions with which to 
approach reform of their mathematics and 
science programs. 

Some messages emerging from analyses 
of TIMSS data have received more attention 
than others. For example, recent analyses 
have raised issues about content tracking and 
the role of socioeconomic factors in students' 
access to a rigorous curriculum.” In order to 
address this issue, states, districts, and 
schools will need to look at a range of issues 
associated with course assignment, teacher 
assignment, professional development, and 
system incentives. While TIMSS data have 
highlighted this problem, it remains to be 
seen whether jurisdictions will take the com- 
prehensive steps necessary to address it. 

Several Benchmarkers have been able to 
complete more extensive analyses of their 
TIMSS-1999 data. These analyses were 
undertaken in cooperation with external 
partners such as Boston College, Michigan 
State University, and the North Central 
Regional Educational Laboratory. Bench- 
markers engaged in these kinds of partner- 
ships include SciMathMN, First in the World 
(FITW), Michigan Invitational Group, and 
the Math and Science Collaborative. These 
analyses examined the complex relationships 
between student achievement in mathemat- 
ics and science and various educational 
inputs such as teaching practices, profession- 
al development, socioeconomic status, and 
student access to a rigorous curriculum.’^ 

General messages emerging from TIMSS- 
1995 data have thus far had a greater influ- 
ence on shaping particular reform initiatives 
in our sampled Benchmarking jurisdictions 



than TIMSS-1999 Benchmarking data. This 
may be partly a function of the broader scope 
of the TIMSS-1995 data and the intense 
efforts related to its analysis and dissemina- 
tion. It may also reflect the difficulty jurisdic- 
tions have had analyzing and acting on com- 
plex local data. In those jurisdictions that 
have recently completed extensive analyses 
of their TIMSS-1999 data, it is also too soon to 
know whether and how these analyses will 
impact policy and practice. 

Responding to TIMSS: Local 
Improvement Efforts 

TIMSS data raised questions primarily 
about what is taught and how it is taught. 
These questions led to reform efforts focused 
on curriculum and professional develop- 
ment. In this section, we review the changes 
to policy and practice implemented by 
Benchmarkers in their attempts to improve 
instruction and achievement. While TIMSS 
provided an excellent starting point, it could 
not offer comprehensive solutions. Bench- 
markers combined TIMSS data with state and 
local assessment data and their knowledge of 
exemplary materials and best practices to 
craft appropriate solutions to TIMSS-generat- 
ed questions. Benchmarkers' ability to use 
TIMSS data was thus influenced by their 
existing capacities to analyze and respond to 
data and take advantage of available 
resources. 

Developing and Revising Standards 

Several Benchmarkers used TIMSS-1995 
data to launch their own projects aimed at 
providing mathematics and science content 
standards as a template for district curricular 
reform. For example, mathematics and sci- 
ence teachers and curriculum specialists par- 
ticipating in one of FITW's Teacher Learning 
Networks’^ developed their own grade-level 
content standards after reviewing a complete 
TIMSS curriculum analysis, international and 
local curricula, and state standards. In addi- 
tion, they consulted an analysis of national 
content standards and the curricula of two 
high-achieving TIMSS countries conducted 
by Mid-Continent Research for Education 
and Learning.’^’ 
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TIMSS-1995 data were released and ana- 
lyzed at a time when many state legislatures 
and departments of education were debating 
and implementing new content standards. In 
many cases these standards are quite broad 
in each subject area. Several Benchmarkers 
made use of TIMSS results in attempts to 
encourage more focused and rigorous con- 
tent standards. The Math and Science Collab- 
orative viewed the proposed Pennsylvania 
standards as "very much clay on the table" 
and were concerned that the standards 
should reflect lessons from TIMSS as well as 
national standards. They developed opportu- 
nities for educators to look at the proposed 
standards for rigor and coherence; findings 
from the resulting analysis were presented to 
the state. The Collaborative was also asked to 
testify before the State Senate when it was 
conducting a final review of science stan- 
dards. The SMART Consortium used TIMSS 
to help in the development of Consortium- 
wide content standards. These standards 
were used as a resource by the Ohio Depart- 
ment of Education to inform its own mathe- 
matics and science standards. 

Curriculum Review 

Benchmarkers made extensive use of 
TIMSS data in evaluating and revising their 
mathematics and science curricula. General 
lessons from TIMSS-1995 national-level data, 
analysis of international curriculum vari- 
ables, and information about their own stu- 
dents' strengths and weaknesses gave Bench- 
markers a number of specific strategies and 
goals with which to approach these tasks.’^ 
For some Benchmarkers, TIMSS findings 
prompted their first efforts to develop sys- 
tem-wide coherent and rigorous curricula. 

FITW helped member districts to conduct 
their own curricular analyses. This effort was 
guided by use of the TIMSS General Topic 
Trace Map (GTTM), a framework consisting 
of 44 mathematics and 79 science topics 
(Schmidt, Raizen, Britton, Bianchi, & Wolfe, 
1997; Schmidt, McKnight, Cogan, Jakwerth, 
& Houang, 1999). Using the GTTM, districts 
could assess the rigor and coherence of their 
curricula based on the breadth, duration, and 
flow of topics.^^ Districts could use these 
analyses in conjunction with other data to 
raise important questions about their curricu- 
la.*^ In one FITW district, for example, the 







District Action Team used the results of the 
FITW curriculum analysis and its own 
assessments to make a general plan for cur- 
ricular reform. This plan was referred to a 
mathematics and science curriculum com- 
mittee composed of teachers, administrators, 
and central office staff, who revised the cur- 
riculum to make it more rigorous and coher- 
ent, especially in eighth-grade mathematics. 
They purchased new instructional materials 
to reflect these changes and are now in the 
process of implementing the new curriculum. 

Naperville School District 203 also used 
the TIMSS data in development of their 
revised mathematics curriculum and are cur- 
rently using it to inform revision of their sci- 
ence curriculum. In their recent revision of 
the mathematics curriculum, a curriculum 
committee looked at TIMSS-1995 videos, con- 
sulted research, and made use of lessons 
regarding the breadth and lack of depth and 
rigor of curricula in the United States. In 
response, they reduced the number of topics 
covered at each grade level and purchased 
multiple resources rather than single text- 
books to implement the curriculum. 

Teachers and administrators involved in 
curriculum reform efforts reported great ben- 
efits to their own understanding of content 
and pedagogy. One example comes from the 
Math and Science Collaboratives's Curricu- 
lum Framework Project. This project con- 
vened teacher-leaders from across the region 
to develop a framework that could be used 
by district teams of teachers and administra- 
tors in reforming their own curricula. Based 
on Schmidt's analysis of TIMSS curriculum 
variables, the project's goal was to ensure 
that the framework reflected international 
content standards and covered five-to-seven 
"big ideas" at each grade level. A participat- 
ing teacher called this "the best professional 
development experience I've had in my 
whole career." After working on the frame- 
work, she used it in her own district to design 
a six-week curriculum reform effort in math- 
ematics. Participants focused on articulating 
concepts and content across grade levels and 
developing performance assessments. With 
the Collaborative's help, participants later 
purchased materials to help them implement 
their new curricular goals. 
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Programs and Materials 

Benchmarkers mentioned lessons from 
TIMSS as a significant influence on their 
choice of curriculum materials. In efforts to 
focus on fewer topics in greater depth, many 
districts have purchased materials to accom- 
pany units or modules, either as a supple- 
ment or a replacement for textbooks. The 
Math and Science Collaborative provides 
support to districts in their selection of new 
materials by sponsoring workshops and 
maintaining a clearinghouse of exemplary 
materials that reflect rigorous national and 
international standards. Based on TIMSS 
findings, they began an initiative to encour- 
age the use of these materials and greater cur- 
ricular focus at the middle school level. As 
part of a separate initiative to enable all stu- 
dents to have access to rigorous curriculum 
at the high school level, they are also spon- 
soring implementation of the Cognitive Tutor 
Project, a curriculum program developed at 
Carnegie Mellon University.’® 

Professional Development 

While Benchmarkers varied in the degree 
of professional development opportunities 
they sponsored, all acknowledged its critical 
role in building the capacity to both analyze 
and use TIMSS data. As a representative from 
FITW put it, "If you don't build the capacity, 
it makes no difference what the TIMSS data 
say; you're not going to get anywhere." Pro- 
fessional development efforts were generally 
focused on curriculum review, implementa- 
tion of new curricula, or on issues of peda- 
gogy raised by TIMSS analyses. 

Building District Capacity 

States and consortia recognized that 
instructional improvement hinges ultimately 
on local efforts. In most cases, districts are 
responsible for providing the kind of ongo- 
ing support necessary to help administrators 
and teachers make meaningful and lasting 
changes to school and classroom practice. 
Many professional development efforts 
therefore focused on helping district leader- 
ship teams think about how best to imple- 
ment lessons from TIMSS. 

The Math and Science Collaborative 
sponsored a Core Leadership Training for 
district teacher and administrator teams to 



use TIMSS data relevant to curriculum, 
instruction, and assessment as a tool for 
improving their mathematics and science 
programs. These teams looked at national 
and local TIMSS data and analyses and com- 
pleted online challenges with TIMSS- 
released items. They then identified impor- 
tant areas of action in their own districts and 
developed associated professional develop- 
ment plans. In one district, a participant 
developed a summer workshop for math and 
science teachers to expose them to important 
curriculum and pedagogy issues raised by 
TIMSS. Following this workshop, a Science 
Planning Group redeveloped the district's 
elementary science program, trimming con- 
tent and increasing focus on student inquiry 
and problem solving. The team also devel- 
oped associated performance assessments 
and replaced textbooks with exemplary 
materials that reflected the new curriculum. 
A Math Leadership Team developed a two- 
year professional development program 
focused on helping teachers change their 
pedagogy away from simply teaching algo- 
rithms toward a more problem-solving 
approach. 

In Minnesota, SciMathMN developed a 
series of workshops designed to help district 
leadership teams use data to make continu- 
ous improvements in their mathematics and 
science programs. In these workshops, they 
used Minnesota TIMSS data and TIMSS- 
1995-related professional development mate- 
rials’’ to help district administrators and 
teachers think about the kinds of questions 
raised by data and how these inquiries might 
lead to system improvement. For many dis- 
tricts, the data raised questions about cur- 
riculum sequences and materials; TIMSS data 
prompted conversations across grade levels 
that in some cases had never occurred. Oth- 
ers looked at pedagogical practices based on 
issues raised by analyses of TIMSS video 
data.“ The major challenge presented by 
these workshops was getting district leader- 
ship teams to take a comprehensive look at 
their mathematics and science programs 
rather than just focusing on one aspect such 
as pedagogical style or middle school cunic- 
ula. Districts do not always have sufficient 
personnel or other resources to engage in this 
kind of comprehensive effort. 
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Building Teacher Capacity 

Shaping Teachers' Professional Devel- 
opment Options. A number of Benchmarkers 
are using TIMSS to guide the range of profes- 
sional development experiences they provide 
for teachers. Based on data demonstrating 
the relative weakness of teachers' content 
knowledge in the United States compared to 
the content knowledge of teachers in high- 
achieving nations, they are increasing the 
number of content-related professional 
development opportunities for teachers. 
These can take the form of sunruner institutes, 
graduate courses, or school-based support 
from content specialists. Other TIMSS-related 
professional development offerings address 
pedagogical strategies related to teaching for 
understanding and inquiry-based learning. 

The SMART Consortium, for example, 
has developed a range of professional devel- 
opment offerings that can be used by districts 
to help teachers implement new curricula 
and improve their pedagogical techniques. 
Responding to their own TIMSS data which 
suggested a relationship between teachers' 
content knowledge and student achievement, 
they also developed a series of content- 
focused sunruner institutes for teachers. To 
assess the need for this initiative, they sup- 
plemented their TIMSS data by developing 
an additional instrument to measure teacher 
content knowledge. 

Lesson Study. Analysis of TIMSS video- 
tape data presented in Stigler and Hiebert's 
(1999) widely disseminated book. The Teach- 
ing Gap: Best Ideas From the World's Teachers for 
Improving Education in the Classroom, has 
focused attention on the great need to sup- 
port development of teachers' knowledge 
and practice. Their analysis found, in partic- 
ular, that teachers in the United States focus 
their instruction less on developing students' 
conceptual understanding than do Japanese 
and German teachers. They also described 
"lesson study," a routine component of pro- 
fessional practice in Japan, as a powerful 
means to help teachers develop this capacity. 
Teachers engaging in lesson study plan col- 
laboratively, observe, and critique lessons 
designed to address particular areas of com- 
mon curricular concem.^^ 

Use of lesson study does not follow 
directly from TIMSS data; it is, however, a 



practice used by one of the highest achieving 
TIMSS nations to help teachers develop ped- 
agogical skills that TIMSS data suggest are 
important. As such, it provides an interesting 
example of the way comparative studies can 
provide insights into other ways of approach- 
ing teaching and learning.^ Districts and 
schools across the country, as well as a num- 
ber of Benchmarking jurisdictions, have 
begun to implement lesson study as part of 
their professional development programs. 

Augmenting Existing Initiatives. In 
many cases, Benchmarkers' efforts to analyze 
data and reform curricula are directly linked 
with their existing strategies to build profes- 
sional capacity at all levels. For example, 
Naperville's extensive use of data is facilitat- 
ed by its historic and continuing commitment 
to professional development and collabora- 
tive decision making. Curriculum revision in 
Naperville is undertaken by 40-member cur- 
riculum committees that follow seven-year 
cycles of review, development, and imple- 
mentation. These committees include teacher 
representatives from every school in the dis- 
trict. Leadership teams within these commit- 
tees are given extra support by the district to 
develop their knowledge of research, reform 
strategies, and available curriculum materi- 
als. They are sent to national, state, and local 
conferences and given release time within the 
school year to examine research and best 
practices. During the implementation phase 
of a revised curriculum, all system teachers 
attend a curriculum workshop in the summer 
and are provided with release time during 
the school year to develop collaborative 
units, attend grade-level meetings, and par- 
ticipate in continuing professional develop- 
ment opportunities. 

The Delaware Science Coalition and 
Miami-Dade County incorporated TIMSS 
results in the extensive site-based profession- 
al development support systems they had 
established, supported largely by grants from 
the National Science Foundation. In Miami- 
Dade County, educational specialists work- 
ing in the district's Urban Systemic Program 
use a TIMSS-derived curriculum mapping 
process (as described above in relationship to 
FITW) to help teachers and administrators 
make data-driven decisions using their state 
science assessment. In Delaware, the Local 
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Systemic Change Initiative involves educa- 
tors across the state in a collaborative process 
for reviewing curriculum, selecting instruc- 
tional units, purchasing or developing mate- 
rials, providing associated professional 
development opportunities, and developing 
assessments. Influenced partly by TIMSS 
data. Leadership Teams have chosen to 
emphasize inquiry-based learning and to 
focus on fewer topics in greater depth.^ 

Evaluating Efforts 

TIMSS is clearly a useful tool in guiding 
reforms according to international standards 
of achievement and instruction, but it is as 
yet unclear whether and how TIMSS- 
inspired reforms achieve their goals. As 
reported above, TIMSS has been used widely 
to highlight important issues and to suggest 
practices that may be successful in improving 
mathematics and science programs. As with 
any use of data to drive reform, these are 
important first steps but do not ensure sys- 
tem improvement. It is critical that states and 
districts rigorously evaluate the impact of 
their reforms using indicators that provide 
valid and continuous data on larger system 
goals. State and local assessments vary in 
their ability to measure progress according to 
the international standards represented by 
TIMSS and, in some cases, comprehensive 
assessment programs in mathematics and 
science have yet to be implemented. Further, 
TIMSS raises questions about the quality of 
curriculum and instruction that cannot be 
answered by assessment data alone. 

Assessments 

In the context of standards-based reform, 
assessments aligned with learning standards 
and achievement goals are a critical compo- 
nent of effective evaluation. TIMSS has been 
used by a number of Benchmarkers as a tool 
with which to examine state and district 
assessments for appropriate content and scal- 
ing. For example, the Delaware Science 
Coalition has made extensive use of TIMSS in 
ensuring that Delaware's state test, the 
Delaware Student Testing Program (DSTP), is 
aligned with international standards. The 
Coalition is also using TIMSS in a project 
designed to make a conceptual framework of 
Delaware's science achievement and assess 
whether the DSTP is measuring what they 



want it to measure. Initial results indicate a 
greater need to focus on problem solving 
over data retrieval skills. Academy School 
District #20 in Colorado was able to use its 
TIMSS results to put student achievement in 
mathematics on the Colorado Student 
Assessment Program (CSAP) in context. Scal- 
ing on the CSAP indicated that only 25% of 
their students had achieved a proficient rat- 
ing, whereas results from TIMSS-1999 indi- 
cated that their students were in the top quar- 
tile of TIMSS-1999 nations and jurisdictions. 

In most cases, state assessments get more 
attention from districts and schools than 
TIMSS as they are directly linked with 
accoimtability systems and consequences. 
When these tests are aligned in form and con- 
tent with TIMSS, they can be effective tools to 
measure progress toward TIMSS-inspired 
system goals. When the tests differ signifi- 
cantly from TIMSS, however, systems that 
implement them in conjimction with TIMSS- 
inspired reforms may be sending conflicting 
messages to educators. In one district, for 
example, there is significant pressure to per- 
form well on the state science assessment 
which covers a broad range of topics at each 
grade level. This makes it difficult to focus in 
depth on fewer topics as do the higher 
achieving TIMSS nations. In one state, legis- 
lators recognized the importance of raising 
mathematics achievement to global stan- 
dards yet voted to implement a graduation 
test that focused more on low-end basic skills 
than the more rigorous content tested in 
TIMSS. It can also be difficult for district per- 
sonnel to focus on lessons learned from 
TIMSS in the face of constant and increasing 
demands from state and district assessments. 
In one mid-sized district, for example, a sin- 
gle person is responsible for coordinating dis- 
trict activity aroimd curriculum, professional 
development, and a full battery of state and 
district assessments in science. 

Additional Indicators 

Many Benchmarkers recognized the need 
for additional data to inform their decision 
making and to monitor progress toward 
achieving system goals. The Math and Sci- 
ence Collaborative, for example, created Dis- 
trict Learning Profiles that record the per- 
centage of graduating seniors that have had 
success in key gateway courses such as Alge- 
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bra I and upper-level math and science cours- 
es. They are also helping districts to use a 
number of additional tools such as in-class 
and performance assessments to inform and 
monitor their improvement efforts. FITW 
physics teachers launched their own study 
following lower-than-expected results in the 
TIMSS 12th-grade physics exam. These 
teachers reviewed TIMSS results as well as 
their own curricular and instructional prac- 
tices. They supplemented their existing 
knowledge about teachers' beliefs and atti- 
tudes by giving the TIMSS teacher survey to 
a sample of high school teachers (only fourth- 
and eighth-grade teachers were included in 
the original survey).^^ 

In cooperation with the United States 
TIMSS National Research Center at Michigan 
State University, one county in Michigan is 
currently using TIMSS methodology to col- 
lect extensive data on its own student 
achievement and curriculum variables. 
Going beyond TIMSS-1995 and TIMSS-1999 
studies, they are testing every student in 
grades 3-12, and administering a modified 
version of the teacher questionnaire. Analysis 
of these data will be a first step toward 
reform of their mathematics and science pro- 
grams. A major goal of their analysis will be 
exploration of the relationship between 
cohort achievement growth and curriculum.^ 

Conclusion 

The TIMSS experience substantially 
broadened what was described by many 
Benchmarkers as their previous, isolated 
efforts at reform. Comparative achievement 
data allowed jurisdictions to rank their per- 
formance by global standards and to look to 
high-achieving nations and Benchmarkers 
for examples of policies and practices that 
might account for student success, while con- 
textual variables provided broad conclusions 
about the particular practices of higher 
achieving nations. TIMSS has thus been a 
critical tool in highlighting important areas of 
reform. However, it offers no panaceas guar- 
anteed to be effective across contexts and 
over time. The continuing effectiveness of 
TIMSS-inspired reforms hinges ultimately on 
the ability of states, districts, and schools to 
craft local solutions and to evaluate and 
refine their efforts. 



Comprehensive, effective use of TIMSS 
data requires coordinated attention to issues 
of curriculum, assessment, instructional 
materials, professional development, teacher 
qualifications, and student access to rigorous 
curriculum, among others. Capacity to 
address these issues varied between Bench- 
marking jurisdictions as well as within differ- 
ent components of these jurisdictions. Bench- 
markers with existing personnel and infra- 
structure devoted to these issues were better 
situated to take advantage of TIMSS data 
than others. Capacity varied within jurisdic- 
tions as well: within states and consortia, for 
example, some districts had greater existing 
system resources than others; many Bench- 
markers mentioned the greater difficulty of 
implementing TIMSS-related changes at the 
high school level, in keeping with their exist- 
ing difficulties influencing instruction in high 
schools. These differences reinforce emerging 
questions about whether and how data-dri- 
ven strategies for reform can help to increase 
performance in low-performing schools and 
systems; unless data are accompanied by 
efforts to build capacity for their analysis and 
use, they may be unlikely to address differ- 
ences in student achievement resulting from 
existing differences in system capacities. 

TIMSS has clearly succeeded in both 
focusing attention on and guiding improve- 
ment of mathematics and science instruction 
in many Benchmarking jurisdictions. Bench- 
markers' experience illustrates the important 
role that rich, comparative data can play in 
reform efforts; it also indicates the limited 
power of data in the absence of system capac- 
ity to analyze them and to craft, implement, 
and evaluate reforms. In learning from the 
TIMSS experience, it is thus important to con- 
sider the particular qualities of the TIMSS 
data as well as the factors that have con- 
tributed to Benchmarkers' ability to analyze 
and use them. 
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End Notes 

1. In this Policy Brief, "TIMSS" refers to 

both TIMSS-1995 and TIMSS-1999. For an 
overview of TIMSS-1995 and TIMSS-1999 
data collection and results, consult; 
http://nces.ed.gov/ timss. While both 

TIMSS-1995 and TIMSS-1999 provide inter- 
national comparative student achievement 
data as well as information on numerous con- 
textual variables, they differ from each other 
in several ways. Differences relevant for this 
discussion include: 1) TIMSS-1995 collected 
data on student achievement in the fourth 
and eighth grades and the final year of sec- 
ondary school whereas TIMSS-1999 collected 
data only in the eighth grade; and 2) the 
TIMSS-1999 Benchmarking study allowed 
jurisdictions within the United States to par- 
ticipate independently. 

2. Two jurisdictions participated in exten- 
sions of the TIMSS-1995 data collection. 

3. Jurisdictions were also chosen based on 
stated commitment to using TIMSS data. Our 
sample is therefore not necessarily represen- 
tative of all Benchmarkers' experiences. 

4. For more information on these contex- 
tual variables and the TIMSS data collection 
in general, consult http://timss.bc.edu/ 
TIMSSl/abou t_main.html. 

5. A number of reports supported TIMSS 
data dissemination. ITiese include: summary 
reports by the National Center for Education 
Statistics, detailed descriptive analyses by the 
International Study Center at Boston College, 
analyses of curriculum variables by the 
TIMSS Resource Center at Michigan State 
University, and analyses of videotape data by 
LessonLab at the University of California at 
Los Angeles. Associated materials such as the 
TIMSS Resource Kit (Office of Educational 
Research and Improvement, 1999) highlight- 
ed major TIMSS findings and provided tools 
for local reform. 

6. Each provide a limited amount of data 
over time. TIMSS-1995 tested representative 
samples of third- and fourth-grade and sev- 
enth- and eighth-grade students. TIMSS-1999 
tested a cross-sectional sampling of eighth- 
grade students four years after a cross-sec- 
tional sampling of fourth-grade students 
were tested. In addition, the First in the 
World (FITW) Consortium has collected two 



sets of data, one as a TIMSS follow-up in 1996 
and one in TIMSS-1999. 

7. Several jurisdictions, including Min- 
nesota and FITW, used TIMSS-1995 method- 
ology to collect their own data. 

8. The Math and Science Collaborative is 
the organization that sponsored participation 
of the Southwest Pennsylvania Consortium 
in the TIMSS-1999 Benchmarking Study. 

9. See, for example, the evaluation of high 
school physics undertaken by FITW physics 
teachers explained later in this Brief. 

10. These relationships have since been 
more fully explored in Schmidt (2001). 

11. See, for example, Schmidt, Kogan, and 
Wiley (2002) and the analysis of Southwest 
Pennsylvania data at http://www.msc.col- 
laboratives.org. 

12. For more details about these analyses 
consult the following web sites: 

The Math and Science Collaborative: 
http://www.msc.collaboratives.org 

First in the World: http://www.ncre!. 
org/ re/ ae2 

SciMathMN: http://www.negp.gov/ 

reports/ mntimss.pdf 

Michigan Invitational Group: http:// 
mathematicallysane.com/ evidence/ mig.asp. 

13. Teacher Learning Networks are at the 
core of FITW's overall strategy of building an 
inter-district learning community. For more 
information about these networks, see 
www.ncrel.org/ re/ tlnr. 

14. To find out more about FITW's stan- 
dards project, consult http://www.ncre!. 
org/re/fitwsp/. 

15. Some Benchmarkers are also making 

use of the TIMSS 2003 Assessment Frame- 
works and Specifications. These are available 
at http:/ / isc.bc.edu/ timss2003i/frame- 

work.html 

16. The working manual is available at 
http:/ / www.ncrel.org/ re/ gttmr. 

17. This process of curriculum analysis is 
detailed in North Central Regional Educa- 
tional Laboratory (2002). An associated tool 
for curriculum analysis can be accessed at 
www.ncrel.org/currmap. 
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18. For information about this program, 
consult http://www.carnegielearning.com. 

19. For example, National Research Coun- 
cil (1999). 

20. The most common resource in this 
effort is Stigler and Hiebert (1999). Frequent 
use is also made of released videotape data. 
Information available at http://nces.ed.gov/ 
timss/timss95/ video.asp 

21. In addition to Stigler and Hiebert's 
work, a number of scholars have helped to 
explain and popularize lesson study in the 
United States. See the references at the end of 
this Brief. 

22. Another example, not described in this 
Brief, is the use of Singapore's Mathematics 
Textbooks. 

23. This initiative first focused on elemen- 
tary science but has expanded to include K- 
12 mathematics and science. 

24. For additional information on this pro- 
ject, consult http:/ / www.ed.gov/ pubs/ 
FirstLook/ImproveMath.html 

25. At the time of publication, results from 
this analysis were not yet available. For addi- 
tional information, contact the U.S. TIMSS 
National Research Center at http:// 
ustimss.msu.edu. 
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Additional TIMSS-related Resources 

General 

International Study Center at Boston College TIMSS Homepage: http:/ / timss.bc.edu/ 

National Center for Education Statistics TIMSS Homepage: http:/ / nces.ed.gov/ timss/ 

National Research Council. (1999). Global perspectives for local action: Using TIMSS to improve US. mathematics and sci- 
ence education. Washington, DC: National Academy Press. 

Curriculum 

Curriculum mapping tool developed by the TIMSS National Research Center and the North Central Regional Edu- 
cational Laboratory: http://currmap.ncrel.org/default.htm 

TIMSS 2003 Assessment Frameworks and Specifications: http:/ /isc.bc.edu/timss2003i/framework.html 
U.S. TIMSS National Research Center at Michigan State University Homepage: http://ustimss.msu.edu/. 

Videotape Study 

Lesson Lab at the University of California-Los Angeles: http://www.lessonlab.com/index.htm 

Stigler, J. W, & Hiebert, J. (1999). The teaching gap: Best ideas from the world's teachers for improving education in the class- 
room. New York: Free Press. 

TIMSS 1995 Video Study Homepage: http://nces.ed.gov/timss/timss95/video.asp 

Lesson Study 

Lesson Study Research Group at Columbia University Teachers' College: http://www.tc.columbia.edu/lesson- 
study/ 

Lewis, C., & Tsuchida, I. (1998). A lesson is like a swiftly flowing river: Research lessons and the improvement of 
Japanese education. American Educator, Winter, 14-17 & 50-52. 

Research for Better Schools lesson study resource guide: http://www.rbs.org/lesson_study/index.shtml 

Yoshida, M. (1999). Lesson study ljugyokenkyu] in elementary school mathematics in Japan: A case study. Paper presented 
at the annual meeting of the American Educational Research Association, Montreal, Canada. 
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Benchmarking Jurisdictions 



The following 10 Benchmarking Jurisdictions were 
contacted for this study: 

Academy School District #20 
7610 North Union Boulevard 
Colorado Springs, CO 80920 
Contact: Alisabeth Hohn 
Phone: (719) 598-9534 
E-mail: ahohn@d20.co.edu 

Delaware Science Coalition 

Delaware Foundation for Science and Mathematics 

Education 

Room 1115, Community Service Building 

100 West 10th Street 

Wilmington, DE 19801 

Contact: Jack Collette 

Phone: (302) 397-0034 

E-mail: jackcollette@dca.net 

First in the World Consortium 

1250 Sanders Road 

Northbrook, IL 60062 

Contact: David Kroeze 

Phone: (847) 498-2610 

E-mail: kroeze.d@northbrook27.kl2.il.us 

Math and Science Collaborative (Southwest Pennsyl- 
vania Consortium) 

2650 Regional Enterprise Tower 
425 Sixth Avenue 
Pittsburgh, PA 15219 
Contact: Nancy Bunt 
Phone: (412) 201-7404 
E-mail: buntn@collaboratives.org 

Miami-Dade County Public Schools 

1500 Biscayne Boulevard 

Suite 327-T 

Miami, FL 33132 

Contact: Kathryn Carr 

Phone: (305) 995-1916 

E-mail: kcarr@sbab.dade.kl2.fl.us 



Missouri 

Missouri Department of Education 
205 Jefferson Street 
P.O. Box 480 

Jefferson City, MO 65102-0480 
Contact: James Friedebach 
Phone: (573) 751-1395 
E-mail: jfriedeb@mail.dese.state.mo.us 

Naperville Community School District 203 

203 Hillside Road 

Naperville, IL 60540 

Contact: Jodi Wirt 

Phone: (630) 420-6319 

E-mail: jwirt@ncusd203.org 

Rochester City School District 
131 West Broad Street 
Rochester, NY 14614 
Contact: Michael Chan 
Phone: (716) 262-8135 
E-mail: michael.chan@rcsd-kl2.org 

SciMathMN 

1500 Highway 36 West 

Roseville, MN 55113-4266 

Contact: Bill Linder-Scholer 

Phone: (651) 582-8813 

E-mail: bill.linder-scholer@state.mn.us 

Project SMART Consortium 
22800 Cedar Point Road 
Cleveland, OH 44142 
Contact: Terry Krivak 
Phone: (440) 962-3094 
E-mail: terrykrivak@oai.org 




U.S. Department of Education 
Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 
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